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Abstract. With a significant increase in the number of e-learning resources the issue of quality 
is of current importance. An analysis of existing scientific and methodological literature shows 
the variety of approaches, methods and tools to evaluate e-learning materials. This paper proposes 
an approach based on the procedure for estimating parameters of local factors and receiving the 
integral index of usability quality of e-learning modules. We present a mathematical model which 
serves as a basis for the automated procedures for expertise. The use of fuzzy logic allows to reduce 
greatly the complexity of evaluating the formation of a repository of e-learning modules. 

The proposed approach is focused on the situation, when the university 

— has amassed a large number of e-learning modules that have to be assessed in terms of 
ergonomics; 

— is able to use experts in ergonomics and organization of e-learning (the experts can provide, 
as a rule, qualitative assessment); 

— is limited in resources on the development of special software for evaluation of e-learning 
modules; 

— is forced by the need to reduce the cost of expertise to be limited to considering only the main 
quality indicators that have the greatest impact on the ergonomics of e-learning modules. 

For automation of the ergonomic examination procedures a MatLab system is used, in particular 
Fuzzy Logic Toolbox. 

Application of the well-known mathematical tools and widely used means of processing expert 
qualitative assessments can significantly reduce the cost of the expertise. 

Keywords: e-learning module, expertise, usability evaluation, fuzzy logic. 


1. Introduction 

The use of information technology is an integral part of modern education. 

In terms of active use of information technology in education e-learning resources are 
dynamically developing. A variety of commercial companies (e.g., Microsoft, Harbinger 
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Knowledge Products, Algoryx, Technomatix) are involved in the development of e- 
learning. This process also involved higher educational institutions (for example, Mas¬ 
sachusetts Institute of Technology, Monterey Institute for Technology and Education, 
The Open University and The Russian National Open University “INTUIT”). 

In Ukraine, Sumy State University (SSU) is one of the leaders in the development of e- 
learning modules of the unified educational e-learning system of the university (Vasylyev 
et al., 2007; Lavryk et al, 2007). Among the SSU innovations there are various types of 
e-learning modules: an electronic synopsis, a test, a computer tutor, a computer learning 
game, a virtual lab (http : / / fpkpi . sumdu. edu. ua/en/nav-main-en. html). 

The developed e-learning modules are related to the following areas: humani¬ 
tarian, natural science, mathematics, economic, medical, and engineering. The link 
http://fpkpi.sumdu.edu.ua/en/int_tool.html refers to the some e- 
learning modules examples such as the virtual lab “Analog hardware complex for simulat¬ 
ing dynamical systems on their structural models”, biochemistry laboratory-based work 
“Method of Iyendrashek”, virtual operation “Aortocoronary Bypass”, English language 
learning game. 

At present there are widely spread open educational resources which can be accessed 
via special repositories. Examples of such repositories are international, national and in¬ 
stitutional repositories like: MERLOT, EducaNext, ARIADNE, EduTube, The National 
Repository of Online Courses, Connexions, The Russian Federal Center of Information 
and Educational Resources. 

A significant increase in the number of developed commercial, public or local re¬ 
sources of e-learning poses a difficult choice to users. One of the decisive factors influ¬ 
encing the choice may be the quality of the resource. For example there might be the 
repositories of the authoritative institutions of higher education. 

2. Problem Formulation 

In the practice the e-learning laboratory of Sumy State University was faced with such 
a situation: 

— on the one hand (estimates vary), one of the largest banks of e-learning modules in 
Ukraine is accumulated at the University; 

— on the other hand, e-learning modules were developed at different times by differ¬ 
ent teachers, without uniform standards and have a different quality. 

The analysis shows that there are alternatives to e-learning modules and the extent 
of their use by students, who have the ability to choose, is very different. Moreover, this 
difference is not so much the content, but the ergonomics. 

The acuteness of the situation was aggravated by: 

— a large number of foreign students (more than 1000); 

— implementation of a number of international projects for open educational elec¬ 
tronic resources. 

Many of the modules to be simply removed from the bank of the electronic resources. 
However, this must be done, on the one hand, by the system of the formal parameters, and 



Organizational Approach to the Ergonomic Examination of E-Learning Modules 109 


on the other hand - the backing-out procedure must have a powerful “explanatory func¬ 
tion” - type of “what modern ergonomic requirements for qualitative e-learning modules 
are”. 

Thus, the problem can be described as follows. It is necessary: 

— to exclude from the learning process the e-learning modules of poor quality in the 
view of ergonomics; 

— to include in the technology of the development of new e-learning modules 
a mandatory step of acceptance, which in its part, includes an ergonomic exper¬ 
tise. 

Procedure requirements of the ergonomic expertise: 

— automation of conducting an ergonomic assessment; 

— compliance with existing technologies and the international practice; 

— the ability to communicate with experts in a language approaching natural; 

— the possibility of an integrated assessment of ergonomic quality based on expert 
evaluation of local indicators; 

— the maximum reduction in the complexity of the examination without significant 
loss of its quality; 

— exclusion from consideration groups of parameters, which are of the least signifi¬ 
cant impact on the overall ergonomic evaluation; 

— elimination of the need to purchase (design) special software for examination; 

— the use of widely recognized and available means of automation; 

— the availability of explanation (why making this or that rating). 

Thus, the task is to develop an ergonomic expertise process that meets put require¬ 
ments. 

3. Related Works 

3.1. Usability Evaluation Methods 

Usability as defined in ISO 9241-11:1998 is understood as the “extend to which a prod¬ 
uct can be used by specified users to achieve specified goals with effectiveness, efficiency 
and satisfaction in a specified context of use”. The term “Ergonomics” has a wider mean¬ 
ing and has been introduced in the ISO 6385 in the year 2004: “Scientific discipline 
concerned with the understanding of interactions among human and other elements of 
a system, and the profession that applies theory, principles, data and methods to design 
in order to optimize human well-being and overall system performance”. 

A large set of evaluation techniques exist in the literature. Preece et al. (2007) clus¬ 
tered them in three main approaches: usability testing, analytical evaluation and field 
studies. Usability testing concerns the analysis of users’ performance on the tasks for 
which the system is designed (Preece et al., 2007). This approach has the potential to 
provide reliable results since it involves samples of real users. However, reproducing re¬ 
alistic situations of usage in a laboratory is difficult, e.g., selecting a representative sample 
of users and tasks, training users to master advanced features of the system in a limited 
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time period, or weighting the effect of important contextual factors on their performance 
(Lim etal., 1996). 

Analytical methods are done by usability experts, who put themselves in the intended 
end-users position. Based on the experts expertise and usability heuristics the expert vali¬ 
dates the software (Blecken et al., 2010), and as no user needs to be involved, these evalu¬ 
ation methods fits best early in the development process. Examples of analytical methods 
are “Guidelines”, “GOMS” or “Heuristic Evaluation” (Bernerus A. et al., 2010). The 
analytical approach includes inspections and the application of formal models to predict 
users’ performance. A common inspection method is heuristic evaluation (Nielsen, 1993). 
It involves experts who inspect the system and evaluate the interface against a list of us¬ 
ability principles, i.e., the heuristics. The main advantage is related to cost-saving: they 
“save users” and do not require special equipment or lab facilities (Jeffries and Desurvire, 
1992). In addition, experts can detect a wide range of problems of complex systems in 
a limited amount of time. The main drawback of such a technique is the dependency on 
the inspectors’ skills and experience, as heuristics are often generic and underspecified 
(Doubleday et al., 1997; Law, 2007). 

Field studies differ from the other evaluation approaches because they are conducted 
in natural settings. Their aim is to understand what users do naturally and how technology 
impacts on them (Lanzilotti et al., 2010). They are useful for identifying opportunities for 
new technology, eliciting requirements, deciding how best to introduce new technology, 
and evaluating technology in use. Evaluating usability in the field is difficult, due to the 
complexity of the environment and the activities to be observed, and to the large amount 
of data to be analysed (Pascoe et al., 2000). 

With respect to evaluating the quality of open educational resources in practice there 
are used the method of content reviewing prior to its publication (the experience of MIT 
Open Courseware) or the model of “lenses” for peer evaluation and quality control after 
the open publication (Connexions experience) (Baraniuk, 2008, pp. 232-236). 

An analysis of existing scientific and methodological literature shows the variety of 
approaches, methods and tools for evaluation of e-learning materials. There exist several 
models of evaluation from different perspectives (pedagogical and technical), for example 
Kirkpatrick’s Four Levels of Evaluation (Galloway, 2005), Analytic Hierarchy Process 
approach by Colace et al. (2006), and also Pedagogy Effectiveness Index (Sonwalkar, 
2002 ). 

Oliver et al. (2000, 2007) provide a comprehensive account of the development of the 
field of evaluation of e-learning during the last decade, describing checklists, guidelines 
and toolkits designed to facilitate evaluative work of the academic staff. 

The large number of literature on the subject of quality assessment shows interest 
and the need for more affordable and practical means of evaluation. For example, in the 
materials of the report “Evaluation of e-learning courses” (Jara et al., 2008, p. 12) the em¬ 
phasis is placed on the analysis of the existing evaluation tools specifically for a teacher, 
who is often not an expert in the field of quality assessment. The most common method 
of evaluating the quality of e-learning modules is a method of peer review. Experts are 
offered a variety of questionnaires, checklists in electronic or paper form. 
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Despite several studies have been performed to compare different approaches to us¬ 
ability evaluation, in practice the selection of a specific method is often based on consid¬ 
erations of costs and available resources (Hartson et al., 2003; Law et al., 2009). 

Drawing a conclusion on which method is best is difficult if not impossible task, since 
each method has its strengths and drawbacks and there is no specific method for evalu¬ 
ating e-learning systems. A case study on comparative usability evaluation (Koutsabasis 
et al., 2007) revealed that “no method was found to be significantly more effective or 
consistent that others”. 

Methods of usability evaluation are usually very expensive, time-consuming and their 
results are usually very difficult to analyze. 

From these reasons, there is a need to develop a model of usability evaluation that is: 

— quick, precise and produces results that might be easily analyzed, 

— allowing to get single-value score, 

— able to deal with the users’ language which is full of vague terms, 

— based on mathematical principles. 

3.2. The Advantages of Fuzzy logic 

The approach to usability evaluation based on fuzzy logic are used for Human-Centered 
Systems (Nunes, 2010), web-sites (Gtilfin Buyiikozkan et al., 2010), web-portals (Hub 
et al., 2010), e-learning system (Lanzilotti et al., 2010), quality of education (Valdes- 
Pasaron et al., 2011). We propose to apply this approach to usability evaluation e-learning 
modules. 

Following principles should be implemented in the model of fuzzy usability evalua¬ 
tion (Hub et al., 2010): 

— users do not express the overall score by using numerical values; 

— using their natural language, they evaluate a set of characteristic features that sig¬ 
nificantly affect usability; 

— users’ mental load should be minimized so they can fully focus on the aspects of 
evaluation. 

Usability score is a best approximation of expert knowledge stored in a special 
database. 

4. Problem Solution 

According to the methodology of the ergonomic expertise (Adamenko, 1993) we’ll solve 
the problem as follows: 

1. Determination evaluation criteria. 

2. Screening of options, in which at least one ergonomic factor has a value below 
a certain critical admissibility. 

3. Determination of the integral index of ergonomic performance of e-learning mod¬ 
ules according to set of local indicators. 

Steps 1 and 2 are adequately described in the ergonomic literature (Adamenko et al., 
1993; Rothstein, 1999). The problem of the Stage 3 refers to the classification problem, 
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Fig. 1. Steps of research. 


which can be solved with the involvement of a large number of methods allowing working 
with peer reviews (method of hierarchy analysis, neural networks, fuzzy logic et al.). Due 
to the vagueness of the information, contained in the estimates of experts, we define the 
fuzzy inference method, proposed by Rothstein (1999). 

The usability evaluation process is performed by applying the following steps (Fig. 1). 

4.1. Determination of E-Learning Module Evaluation Criteria 

The degree of study of various factors in literature is investigated in the review Bernerus 
et al. (2010). 

As a result, looking at Fig. 2, we can see that the top factors are general issues of 
usability, such as Navigation, Feedback and User Control. However, it is important to 
note that some general usability factors are more or less important due to the fact that an 
e-learning system is being evaluated. For example the Flexibility and Efficiency of Use 
as well as Visibility of System Status are particularly important for e-learning systems, 
which can also be seen in the result. 

Various studies show the validity of the use of different criteria. So, in the work of 
Giil§in Btiyiikozkan (2010) the author offers the following global criteria: Right and Un¬ 
derstandable Content, Complete Content, Personalization, Security, Navigation, Interac¬ 
tivity, User Interface. 

Costabile (2005) has identified the following groups of criteria: Presentation, Hyper- 
mediality, Application Proactivity, User Activity, Scaffolding, Learning Window. Each 
criterion has a number of attributes. For example, the criterion Learning Window has at¬ 
tributes of organization of a course document, suitability of formats of course document, 
check of assessment test presence, check of usage of communication tools, adequacy of 
learning tools. 

When talking about an e-learning system, we do not distinguish between platform and 
didactic module. Actually, an e-learning platform is a more or less complex environment 
with a number of integrated tools and services for teaching, learning, communicating and 
managing learning material. On the other hand, the didactic module is the educational 
content provided through the platform (the container). Usability attributes for a platform 
generally differ from those of a specific didactic module (content) (Costabile et al., 2005). 
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Fig. 2. Usability factors in e-learning articles. 


The analysis of a number of works allowed to identify 14 basic categories of criteria 
for e-learning modules evaluation. This number determines the high labor intensity for 
the evaluation, therefore it is necessary to identify the most important criteria for the 
general ergonomics. 

For an example of the approach, as a basis, we take a list of criteria categories (Table 1) 
from the review Bernerus et al. (2010). 

4.2. Determination of Criteria Weights for E-Learning Module Evaluation and Creation 
a Criteria Set 

4.2.1. Initial Conditions 

The task of assessing the significance of the criteria can not be solved only once. The 
results of this analysis depend on the conditions of the school, the learning process, the 
contingent of students and other factors. 

The complexity of this step is not only in the organization of the expertise, but also 
in the selection of the qualified experts who know the subject area. The results of such 
expertise in different universities vary. 

For conditions of Sumy State University such examination was organized involving 
3 experts. 
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Table 1 

Usability categories 


Name 

Description 

Error management 

This concerns help and documentation files and how easy it is to find the 
relevant information. 

Leam-ability 

How easy the system is to learn. How long does it take for a user to master 
the system? 

Cognition facilitation, 
recognition & memorability 

Relevant objects, actions and options should be clear. The user should not 
have to remember too much itself. 

Flexibility and efficiency of 

use 

This criteria concerns the possibility for the system to adopt to different 
users with different learning styles and tastes. 

GUI 

This concern not only how “pretty” the interface is but also how logical the 
structure is and how easy it is to read and understand. 

Navigation and exiting 

How easy the system is to navigate and find your way in it. How logical the 
structure is, etc. 

Match between system and 
the real world 

This issue is specifically concerned with logical metaphors and phrases, etc. 

Consistency and standards 

The learner experiences the user interface as consistent (in control, color, ty¬ 
pography, and dialog design) words and concepts that the user can recognise 
and understand. 

Help and documentation 

Help should be easy to search. Any provided help is focused on the learner’s 
task, and lists simple concrete steps to be carried out. 

Accessibility 

How the software can be accessed. All repository access to both teacher and 
learner. 

Learning content design 

Concerns pedagogical aspects in leaming/course materials. Terminology, 
layout and media use are some examples from the papers. 

Visibility of system status 

Concerns system feedback and to the user, both in terms of system status, 
presenting a score or other information and similar. 

User Control and freedom 

The learner can easily turn the application on and off, and can save his user 
profile in different states. 


There are many methods that can be employed to determine weights, such as eigen¬ 
vector, weighted least square, entropy methods and diverse MCDM methods. In our task 
of estimating the ergonomics of the module a subjective notion of comfort is considered. 
Same module with fixed values of the parameters can be interpreted in different ways by 
different experts and, respectively, the degree of comfort will be different. Therefore, it 
is necessary to use methods based on subjective assessments of experts. In this study, the 
most outstanding MCDM approach, AHP (Saaty, 1980) is used to determine the decision 
criteria weights. 

The four step computational procedure is given as follows: 

Step 1. Compare the performance score. The numbers (1, 3, 5, 7, 9) are used to indicate 
the relative strength of each pair of elements. 

Step 2. Construct the comparison matrix. 

Step 3. The consistency ratio (CR) for each matrix is calculated. The deviations from 



Organizational Approach to the Ergonomic Examination of E-Learning Modules 

Table 2 

E-learning modules evaluation criteria weights 


115 


Name Weight 

Error management 0.011 

Leam-ability 0.099 

Cognition facilitation, recognition & memorability 0.067 

Flexibility and efficiency of use 0.184 

GUI 0.1 

Navigation and exiting 0.141 

Match between system and the real world 0.002 

Consistency and standards 0.187 

Help and documentation 0.078 

Accessibility 0.002 

Learning content design 0.123 

Visibility of system status 0.001 


consistency are expressed by the following equation consistency index, and the 
measure of inconsistency is called the consistency index. 

The consistency ratio is used to estimate directly the consistency of pair wise com¬ 
parisons. If CR is less than 0.10, the comparisons are acceptable, otherwise not 
(Saaty, 1980). 

Step 4. The priority weight of each criterion can be obtained by multiplying the matrix 
of evaluation ratings by the vector of attribute weights and summing over all at¬ 
tributes. 

The obtained results are shown in Table 2. 

4.2.2. Using the Results of the Criteria Importance Evaluation 
The significance of the criteria is important at: 

— assessment of the modules (primarily the most important criteria are evaluated that 
allows to postpone a subsequent evaluation of the most important indicators at their 
extremely low values); 

— exclusion from examination the least important criteria (in case of restrictions be¬ 
cause of the complexity of expertise). 

In the next section, as an example, a model for the examination of the most significant 
groups of five criteria is considered: Flexibility and efficiency of use, GUI, Navigation 
and exiting, Consistency and standards. Learning content design. 

4.3. A Model of Ergonomic Expertise Based on Fuzzy Logic 

Suppose we are given a set of local indicators of ergonomic performance of modules 
K = {kj}, j = l,n. Selected indicators of the sets can be separated into some cate¬ 
gories G = { fj,}. i = 1 , m. There is some procedure of conformity evaluation of local 
indicators to some requirements. Given is a set of possible outcomes of usability evalua- 
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tion E = {ei, e 2 , 63 }. Evaluation of the quality of the module is used to make one of the 
following decisions: e\ - conform (module corresponds to the claimed usability recom¬ 
mendations and standards), e 2 - redesign (module not fully corresponds to the claimed 
usability recommendations and standards), e 3 - mismatch (module dose not corresponds 
to the claimed usability recommendations and standards). 

A set of specific parameters, analyzed in each case, depends on many factors. In this 
example for ergonomic examination we restrict ourselves to the following characteris¬ 
tics: Navigation and Exiting, Learning Content Design, Consistency and Standards, GUI 
and Flexibility and Efficiency of Use. This example is illustrative. In reality, reducing 
the number of groups of factors can be held in conjunction with the need to reduce the 
complexity of the examination. This decision is made every time based on the analysis of 
the available time resource of the experts. 

Let E — an integral indicator of the quality of e-learning modules. To evaluate this 
indicator we will use the following information: 

X - Navigation and Exiting, which is evaluated with the following local indicators: 
x± - usability of the keyboard and mouse, x-i - intelligibility and easy to navigate (type 
and location of control keys, key transition points and the path to them, the possibility of 
random and sequential movement of material, etc.), X 3 - usability of the table of contents; 

Y - Learning Content Design, which is estimated with the following local indicators: 
yi - the amount (dosage) of the material on the page (slide), 2/2 - consistency of page 
layout; 

Z - Consistency and Standards, which is estimated with the following local indica¬ 
tors: z\ - readability, z -2 - compliance with the design logic of the text elements (body 
text, headings, subheadings, captions for illustrations, etc.); 

V - Flexibility and Efficiency of Use; 

M - GUI (compliance with the design logic of the objects): m± - ratio aspect, m 2 - 
location, m 3 - color. 

The task of the evaluation is to bring to conformity one of the solutions e±, 62 , £3 with 
the module with known local indicators. 

4.3.1. The Scheme of the Solution 

The general scheme of solving the problem of ergonomic examination of e-learning mod¬ 
ules is shown in Fig. 3 and represents a sequence of actions: 

1. Evaluation of the e-learning module according to the selected indicators on the 
scale of the thermometer. 

2. The procedure of fuzzy inference. 

3. The decision on the conformity of the achieved quality indicators to general and 
special ergonomic requirements and the establishment of the ergonomic quality of 
the module. 

If you decide on the conformity, the module is added to the library e-learning modules 
and can be used for further procedures to select the most suitable module for a particu¬ 
lar user. Otherwise, the recommendations are given for redesign, or discrepancy of the 
module is justified. 
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Fig. 3. The general scheme of solving the problem of ergonomic examination of e-learning modules. 


4.3.2. Fuzzy Inference 

The criteria hierarchy is shown in Fig. 4 as a derivation tree, which corresponds to the 
system of relations: 


E = fe(X, Y, Z, M, V), 

(1) 

X = f x (x i,x 2 ,x 3 ), 

(2) 

y = f y (yi,y2), 

(3) 

z = fz(zi,Z 2 ,Z 3 ), 

(4) 

M = / m (rni,TO 2 ,rn 3 ). 

(5) 


These relations are brought into line with fuzzy logic equations, which allow to define 
the level of index E according to the maximum of membership function: 


H E * {X, Y, Z, M, V) = max { min [fi xir (X) , p Y " ( Y) , p 2 " (Z), 

P- 1,9 



(6) 

p Xj (x!,x 2 ,... ,xi) — max { min [p x * P (xi)] }, 

p=l,ej i=l,Z 

(7) 

F yj (yi: 2 / 2 , • • •, 2/rn) = max { min [(J?**{yi)\ }, 

p—l.Qj i—l,m 

(8) 

F Zj (zi,z 2 ,...,z n ) = max { min_ [p z *" ( z t )] }, 

p=l,hj i=l,n 

(9) 

p Mj (mi,m 2 ,... ,m k ) = max { min [p m * P (rrii)] }, 

p=l,tj i=l,k 

(10) 
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Fig. 4. Relationship of integral and local criteria. 


H Vj (v 1 ,v 2 ,...,v s ) = max { min [p v * P (vi)\ }. 

p—l,Uj i—l,s 


(ii) 


In accordance with Rothstein (1999), the algorithm of fuzzy inference using a gener¬ 
alized derivation tree is: 

1. We fix a vector of values of input variables 

(x{, x* 2 ,.. 2/5, ,z% , t> 2 ,-••,<)• 

2. We determine the values of membership functions of terms-estimates of input vari- 

j p _ _ j p _ _ j p 

ables p x s (x»), p = 1 , 1 , j = l,ej, p y t (yf), p = l,m, j = l,gj, g z < ( z t ), 
i = 1 ,n. 

3. Using (6)—(11), we calculate the membership function of the terms-estimates of 
the output value E, which corresponds to the vector of values of input variables 


(xl,X2, ■ ■ ■ ,x*,yl,y2, ■ ■ ■ ,y^,zf,Z2, ■ ■ ■ ,z*,mt,rri2, ■ ■ ■ ,ml,vf,v 2 ,. 


:)• 


4. We define the evaluation E*, which membership function is maximum: 


y E i {X <Y < Z < M <V) = max [p E i (X, Y, Z, M, V )] 

i=l ,r 

-E = E t. 


( 12 ) 


4.3.3. Evaluation of Local Indicators on the Basis of the Thermometer 
The peculiarity of the local indicators is that they are qualitative in nature, ie do not have 
precise quantitative measurements. Therefore, when evaluating the same figure some ex¬ 
perts may have different opinions. In addition the expert is not always able to estimate 
verbally the local indicator, although intuitively feel the quality. To overcome these dif¬ 
ficulties the local indicators can be estimated on the basis of the thermometer (Fig. 5), 
which is set out in the (Rothstein, 1999) work. 

The beauty of this approach is that different in the sense local indicators are deter¬ 
mined as linguistic variables defined on a single universal set, which is the scale of the 
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Fig. 5. Evaluation of the variable u on the principle of the thermometer. 


Table 3 

Some recommendations to assessment of local quality indicators 


Local indicators 

Levels of assessment of indicators 




Invalid (rejected 
at the prelimi¬ 
nary stage of the 
expertise) 

Minimal 

Average 

Maximal 

Intuitive and easy 
to navigate, X 2 

Non-standard 

location of the 

controls 

Preliminary prepa¬ 
ration before start¬ 
ing work with the 
module is required 

Some comments 

Intuitive interface 

Usability of the ta¬ 
ble of contents xs 

There is no table 

of contents 

In the table of con¬ 
tents there are only 
the main sections 

Some inconve¬ 
nience during the 
transition 

Smooth transition 

between sections 

and subsections 

The amount of ma¬ 
terial on the page, 

Vi 

The redundancy 
of the material on 
the page 

Some redundancy 

Some comments 

A necessary and 
sufficient amount 

of material 

Uniformity of 
pages’ design y 2 

Unreasonable 
stylistic diversity 

Differences in the 
design of some 
pages 

Minor comments 

All pages are in the 
same style 


thermometer. Subjectivity reducing can be achieved through the use of the recommen¬ 
dations in Table 3. For example, the shaded portion of the scale in Fig. 5 for the “build 
quality” corresponds to the high quality of performance. 

4.3.4. Evaluation of the Integral Indicator 

Suppose that the linguistic variables X\ — x$, iji — 'tj2 , z\ — z %, v, m \ — m3 are valued by 
fuzzy terms L - low, M - middle and H - high, which are defined using the membership 
functions (Fig. 6). 

Using fuzzy terms, we define a knowledge of the relations (1-5) in a matrix, fragments 
of which are presented in Tables 4-8. Each group of lines shows a conditional statement 
that links the fuzzy values of the input and output variables. For example, Table 4 shows 
that a statement is the condition of the matching module: 


IF {X = H)&(Y = H)&(Z = H)8z(V = H)&{M = H ) THEN E = ei. 




120 


E. Lavrov et al. 


L M H 



Fig. 6. Membership functions of fuzzy terms. 


Table 4 

A fragment of knowledge of the relations (1) 


X 

Y 

Z 

V 

M 

E 

H 

H 

H 

H 

H 

ei 

M 

H 

H 

M 

H 


H 

M 

M 

M 

H 

e2 

M 

H 

M 

H 

M 


L 

L 

L 

L 

L 

e 3 

L 

M 

M 

L 

M 


Table 5 



Table 6 


A fragment of knowledge A fragment of knowledge 

of the relations (2) of the relations (3) 


XI 

X2 

X 3 

X 

y i 

2/2 

Y 


H 

M 

H 

H 

H 

L 

M 


M 

L 

M 

M 

H 

M 

H 


H 

H 

H 

H 

M 

M 

M 



Table 7 



Table 8 


A fragment of knowledge 

A fragment of knowledge 


of the relations (4) 

of the relations (5) 


21 

22 

Z 


mi 

m 2 

ms 

M 

M 

M 

M 


H 

H 

H 

H 

H 

M 

H 


H 

H 

M 

M 

L 

L 

L 


L 

M 

M 

L 


Fuzzy logic equations, set out in compliance with Tables 4-8, allow to evaluate the 
integral coefficient of agreement for fixed values of local indicators. Using the above 
approach, we form a set of modules that the best meet the requirements. Examples of 
estimates of the three modules are shown in Table 9. 


4.3.5. Automation of the Examination and Analysis of the Approach Benefits 

In Sumy State University examination is carried out using this approach by means of 

MatLab. Basic calculations are performed using Fuzzy Logic Toolbox. This solution has 
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Table 9 

Examples of e-learning usability evaluation 


local 

indicators 

Module 1 

Module 2 

Module 3 

Xi 

^_. 

^_1 

■■1_1 

x 2 

^ 1 

^ 1 

^1 1 

X 3 


^^1 1 

^11 | 

yi 

1 


— ~i 

y2 

^- 1 


■■ 1 

Zi 

i 

■■■■■■_ 

\ _1 





Z2 



] ~1 





m, 



] 1M 1 





m 2 

_ -1 

li^^H^H^lHKZZZZZZZZZZ 

WM. _ _........ZZ 


1 


1 

V 



1 1^ 1 

Decision 

Meets the requirements 

Requires modifications 

Doesn’t meet the requirements 



Fig. 7. Relations between ergonomic quality of e-learning modules, learning content design and GUI. 


the following advantages: 

— no need to purchase (to develop) expensive programs; 

— ease of creating and editing the necessary knowledge bases; 

— no time-consuming calculations; 

— rapid assessment; 

— ability to solve problems, such as “What if?”; 

— focusing on untrained users, user-friendly interface; 

— large graphical representation of data (see example in Fig. 7); 

— ability to connect other tasks (for example, adaptation to the requirements of stu¬ 
dents). 

Some of the automation matters of such assessment are considered by Lavrov and 
Barchenko (2008). 
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5. Conclusions 

The large number of existing scientific and methodological literature on quality assess¬ 
ment of e-learning resource demonstrates the relevance of the theme for the educational 
sector, diversity of approaches, methods and assessment tools, 

The complexity of most methodologies of expertise and lack of available decision 
support systems in the formation of repositories of educational resources make the task 
of developing a procedure for resource evaluation of current importance. 

For the examination of the quality we offer the approach based on the use of fuzzy 
logic, which allows you to: 

— create simple intuitive expert estimation procedures of local indicators of quality; 

— obtain the integral quality assessment; 

— work out recommendations for the development of a repository of educational re¬ 
sources. 

The developed approach is tested in the ergonomic expertise of educational resources 
for distance learning system of Sumy State University. 

The use of models and derived from them automated procedures can significantly 
reduce the complexity of evaluation of educational resources. 

Further studies will be aimed at improving the software tools and techniques for eval¬ 
uation of e-learning modules within the project “Mechanisms of the organization of the 
individual educational trajectories using electronic resources to improve the quality of 
training”, which is scheduled for implementation. 
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Strukturinis metodas ergonominiam el. mokymo moduliij tyrimui 

Evgeniy LAVROV, Olena KUPENKO, Tetiana LAVRYK, Natalia BARCHENKO 

Didejant mokymosi istekliij kiekiui pagrindine problema, kylanti el. mokymo srityje, yra jij 
kokybe. Mokymosi istekliams vertinti mokslineje ir metodineje literaturoje randama daugybe 
skirtingu metodologiji}, metodij ir jrankiij. Straipsnyje pristatomas metodas grjstas lokaliijjij fak- 
toriu parametri} jvertinimu ir el. moduliij panaudojimo kokybes integralaus jvercio gavimu. Pri¬ 
statomas matematinis modelis automatinems funkcijoms realizuoti. Neraiskiosios logikos taiky- 
mas jgalina sumazinti vertinamij el. moduliij kompleksiskuma. Pasiulytas metodas nagrinejamas 
tokioje situacijoje: universitetui pateikiama daugybe el. moduliij. kurie turi buti jvertinti zvelgiant 
is ergonominiij pozicijij; pasitelkiami ekspertai jvertinti el. modulius (ekspertai pateikia taisyk- 
les, kokybinius jvercius); nera specialios programmes jrangos el. moduliij vertinimui; siekiama 
sumazinti kokyb? lemianciij veiksniij skaiciij ir parinkti tik tuos, kurie turi didziausiq jtakq el. 
moduliij ergonomines savybems. „Matlab“ sistema buvo naudojama siekiant jgyvendinti ergono- 
minio tyrimo metodq. Gerai zinomo matematinio jrankio taikymas ir ekspertinis vertinimas leido 
reiksmingai sumazinti tyrimui reikalingas sqnaudas. 



